But the broader TOPIX index rose 0.28 percent to end at a five-year high, helped by gains in banks and brokerages.
Japan Airlines Corp, Asia's biggest airline by revenue, ended down after it posted a first-half loss on Monday and unveiled a restructuring plan that include wage cuts.
The Nikkei finished down 14.36 points at 14,061.60. Last week, the market was closed one day for a holiday and trade was disrupted by a system glitch on the Tokyo Stock Exchange, but the benchmark still gained about 730 points, or 5.5 percent, its biggest weekly percentage rise since August 2003.
"Investors were wary after the Nikkei's rapid rise, as it added 700 points (last week)," said Shinji Igarashi, a manager at Chuo Securities.
"They are also holding back ahead of key economic data," he added, referring to Japanese machinery orders for September on Thursday and gross domestic product for July-September on Friday.
Jun Nishizaki, chief portfolio manager at Nissay Asset Management, said indications that earnings would improve were needed to push the Nikkei higher.
"Judging from reported earnings and current share prices, the Nikkei is already getting too expensive to buy," he said.
The TOPIX closed up 4.13 points at 1,499.12, the highest finish since October 6, 2000.
Konica Minolta, the world's third-largest maker of camera film, plunged 5.3 percent to 957 yen after it said on Friday it swung to a first-half net loss and now expects losses for the full year.
Japan Airlines fell 1 percent to 299 yen after it posted a half-year net loss and warned it now expected to fall into the red for 2005/06. It said it would cut wages by an average 10 percent.
The nation's largest refiner, Nippon Oil Corp, fell 6.3 percent to 914 yen - the worst performer among the Nikkei 225 components - after UBS said lower oil prices could hurt the company's earnings and cut its rating of the stock and its target share price.
In contrast, Inpex Corp jumped 3.6 percent to 847,000 yen and Teikoku Oil Co Ltd added 3.9 percent to 1,220 yen after the two firms said they would merge as a holding company next April.
Bank shares continued to lure buyers. Nikko Citigroup raised its target prices for major banks, citing increased valuation gains from the strong equity market and factoring in a 1 percentage point rise in short-term interest rates.
Sumitomo Mitsui Financial Group rose 1.7 percent to 1.18 million yen while Mizuho Financial Group added 2 percent to 879,000 yen.
In the steel sector, JFE Holdings Inc was up 2.7 percent at 3,760 yen after the world's fourth-biggest steel maker reported a 42 percent jump in first-half profit but kept its annual outlook unchanged.
After the market closed, Daiichi Sankyo Co Ltd, Japan's second-largest drug maker, reported a 1.5 percent rise in first-half profit despite drooping sales of its core cholesterol drug Mevalotin after its patent expired in Japan and Europe, and kept its full-year estimates unchanged. The shares ended down 0.5 percent at 2,150 yen.
Trade was heavy, with 3.69 billion shares changing hands on the Tokyo exchange's first section, not far off the record volume of 3.7 billion shares hit last week.
Advancers beat decliners 893 to 695.